SPANoF: A Scalable and Performant Architecture for NVMeoF-Based Storage Disaggregation with Limited Network Resources

NVMe-over-Fabrics (NVMeoF) is expected to have high-performance and be highly scalable for disaggregating NVMe SSDs High-Speed Network (HSN)-attached storage servers, thus the aggregated in servers can elastically allocated remote host better utilization. However, due well-known connection scalability issue of RDMA NICs (RNICs), RDMA-enabled HSN only provide a limited scale performant Queue Pairs (QPs) NVMeoF I/O queues transfer capsule data between server servers. current implementations, multiplexing multiple onto single RNIC QP not supported yet. In this paper, we investigate how transfers are performed efficiently over with number QPs, propose SPANoF, Scalable Performant Architecture NVMe-over-Fabrics. SPANoF dissolves intrinsic one-to-one mapping relationship allocates dedicated send-list each queue rather than QP, capsules send-lists QP-centric manner remove lock-contention overhead, polls completion notifications interrupt-caused context switch overhead. We implemented Linux kernel evaluated it by FIO benchmarks. Our experimental results demonstrate that avoid performance collapses commercial RNICs QPs system crash domain-specific limited-scale available QPs. Compared native implementation kernel, saturate an three server. lock-based QP-sharing mechanisms, improves bandwidth up 1.55× under 64 KB sequential write requests, throughput 4.18× reduces average latency 28.31% 4 random read requests.